fix(ci): restore torch E2E coverage in unit-vs-E2E summary - #942
Merged
Conversation
PR #898 moved the torch coverage combine step into docker exec without passing COVERAGE_FILE, so the first combine merged .coverage.unit as a parallel shard instead of .coverage_e2e.*. Rename the unit snapshot to .coverage_unit, merge E2E shards explicitly, and warn when coverage_combined.json is missing.
WangLingxun
marked this pull request as ready for review
July 31, 2026 06:39
WangLingxun
requested review from
Xiaoming-AMD,
limou102 and
wenxie-amd
as code owners
July 31, 2026 06:39
- Add branch = true / --cov-branch to unit and E2E .coveragerc so a covered function means both branch arms ran, not just one line. - Only combine E2E shards when present, and require coverage_combined.json to be non-empty, so a failed merge shows red instead of silently succeeding with no file. - Caption the "including all modules" summary line as lines+branches, since coverage.py's percent_covered now folds branches in. - Refresh the stale backends/megatron note: CI E2E now runs 15+ configs/PR, not 1-2.
test_memory_benchmark SIGABRT'd across all 8 ranks in the last run: enabling coverage's branch trace while torchrun's real GPU training subprocesses run is not safe. Unit (a plain pytest process) keeps --cov-branch; only the E2E lanes drop back to line-only coverage.
coverage combine refuses to mix branch data with statement data
("Can't combine branch coverage data with statement data"), so
.coverage_unit (recorded with --cov-branch) can no longer be merged
directly with .coverage_e2e (line-only). Project unit's data down to
its executed-line set into a throwaway file first -- coverage_unit.json
(exported earlier from the original branch-aware data) is unaffected,
and coverage_combined.json is unchanged: still an honest line-only
union of unit + E2E.
Verified locally end-to-end: standalone coverage_unit.json keeps
branch_coverage=true and real missing_branches, while
coverage_combined.json stays branch_coverage=false and correctly
unions both lanes' executed lines.
…e' into fix/ci-torch-e2e-coverage-combine
wenxie-amd
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docker execwithout passingCOVERAGE_FILE..coverage.unitto.coverage_unitso it is not treated as a.coverage.*parallel shard..coverage_e2e.*shards before line-merging with unit data.coverage_combined.jsonto be non-empty, so a failed merge fails the step instead of silently succeeding with no file.--cov-branchfor the unit lane only. E2E stays line-only: turning onbranch = truein.coveragerc_e2emade the trainer subprocess chain crash (SIGABRTacross all 8 ranks intest_memory_benchmark) once coverage's branch trace was active during real GPU training — caught by this PR's own CI run, reverted before merge.coverage combinerefuses to mix branch data with statement-only data (Can't combine branch coverage data with statement data), so.coverage_unit(now branch-aware) can no longer be merged with.coverage_e2edirectly. Project unit's data down to its executed-line set into a throwaway file before merging —coverage_unit.json(exported earlier from the original branch-aware data) still carries real branch info for external tooling, andcoverage_combined.jsonstays an honest line-only union of unit + E2E. Verified locally end-to-end.percent_coverednow folds branches into the unit side of that number.backends/megatronnote incoverage_summary.py: CI E2E now runs 15+ configs/PR, not 1-2.